Chapter 1 

Series and sequences 



Throughout these notes we'll keep running into Taylor series and Fourier se- 
ries. It's important to understand what is meant by convergence of series be- 
fore getting to numerical analysis proper. These notes are sef-contained, but 
two good extra references for this chapter are Tao, Analysis I; and Dahlquist 
and Bjorck, Numerical methods. 

A sequence is a possibly infinite collection of numbers lined up in some 
order: 

Oi, 02, 03, ■ ■ ■ 
A series is a possibly infinite sum: 

Oi 02 03 . . . 

We'll consider real numbers for the time being, but the generalization to 
complex numbers is a nice exercise which mostly consists in replacing each 
occurrence of an absolute value by a modulus. 
The questions we address in this chapter are: 

• What is the meaning of an infinite sum? 

• Is this meaning ever ambiguous? 

• How can we show convergence vs. divergence? 

• When can we use the usual rules for finite sums in the infinite case? 
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CHAPTER 1. SERIES AND SEQUENCES 



1.1 Convergence vs. divergence 

We view infinite sums as limits of partial sums. Since partial sums are 
sequences, let us first review convergence of sequences. 

Definition 1. A sequence (aj)°lo ^■^ ■^'^^'^ e-close to a number b if there 
exists a number N > (it can be very large), such that for all n> N , 

\aj — 6| < e. 

A sequence (aj)°^o is said to converge to h if it is e-close to b for all e > 
(however small). We then write aj —>■ b, or limj^cc c^j = b. 

If a sequence does not converge we say that it diverges. Unbounded 
sequences, i.e., sequences that contain arbitrarily large numbers, always di- 
verge. (So we never say "converges to infinity", although it's fine to say 
"diverges to infinity".) 

Examples: 

• e"" — > as n — > oo, and convergence is very fast. 

• n/(n + 2) 1 as n ^ oo, and convergence is rather slow. 

• (—1)" is bounded, but does not converge. 

• log{n) — CX3 as n — > cx), so the sequence diverges. For a proof that 
log(n) takes on arbitrarily large values, fix any large integer m. Does 
there exist an n such that log(n) > m? Yes, it suffices to take n > e"*. 

Definition 2. Consider a sequence (aj)^o- define the N-th partial sum 
Sn as 

N 

Sn — do + 0,1 + ■ ■ ■ + cbN — ^2 '^i ■ 

We say that the series aj converges if the sequence of partial sums S^ 
converges to some number b as N ^ oo. We then write Yl'jLo % — ^■ 

Again, if a series does not converge we say that it diverges. 
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1.1. CONVERGENCE VS. DIVERGENCE 



Example 1. Consider Y^'^q 2 ^, i.e., 



1 1 1 

This series converges to the limit 2. To prove this, consider the partial sum 



N 



i=o 



Let us show by induction that = 2 — 2^^. The base case N = is true 
since 2^° = 2 — 2^°. For the induction case, assume = 2 — 2^^. We then 
write 

5jv+i = + 2-(^+^) = (2 - 2-^) + 2-(^+i) = 2 - 2-(^+i), 
the desired conclusion. 

Example 2. The previous example was the x — 1/2 special case of the so- 
called geometric series 

l + x + x"^ + x^ + ... 
with a similar argument, we obtain the limit as 

j=0 

provided the condition \x\ < 1 holds. This expression can also be seen as the 
Taylor expansion of l/{l—x), centered at zero, and with radius of convergence 
1. 

Example 3. Consider the so-called harm.onic series 

. 1 1 1 
^+2 + 3 + 4 + --- 

This series diverges. To see this, let us show that the N partial sum is 
comparable to log(A^). We use the integral test 

N I fN + l ^ 

Sn = - > -dx. 

(Insert picture here) 

The latter integral is log(iV+l), which diverges as a sequence. The partial 
sums, which are larger, must therefore also diverge. 
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CHAPTER 1. SERIES AND SEQUENCES 



Example 4. Consider 



oo 



1 



for some q > 0. As a function of q, this is the Riemann zeta function C(?)- 
(A fascinating object for number theorists.) 

We've seen above that q = 1 leads to divergence. A similar integral test 
would show that the series converges when q > 1, while it diverges when 



We now switch to a finer understanding of convergence: certain series are 
absolutely convergent, while others are conditionally convergent. This will 
affect what type of algebraic manipulations can be done on them. 

Definition 3. A series Yl'jLo absolutely convergent ifYl'jLo Wj\ 

converges. If a series is not absolutely convergent, but nevertheless converges, 
we say that it is conditionally convergent. 

The subtlety with conditional convergence is that alternating plus and 
minus signs may lead to convergence because of cancelations when summing 
consecutive terms. 

Example 5. Consider 




This series is not absolutely convergent, because it reduces to the harmonic 
series when absolute values of the terms are taken. It is nevertheless con- 
vergent, hence conditionally convergent, as the following argument shows. 
Assume N is even, and let 



q<l. 




N 



Terms can be grouped 2-by-2 to obtain, for j > 1, 



1 1 1 
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1.1. CONVERGENCE VS. DIVERGENCE 



A fortiori, <j2, so 

N-l 



j=l,3,5,... ^ 

which we know converges. If on the other hand N is odd, then Sn — S^^i + 
Both terms Sn and 1/{N + 1) are converging sequences, so their sum 
converges as well. This proves convergence. 
Note that the series converges to 

11111 , , , 

+ + + ... = log(2). 

2 3 4 5 6 

This is the special case x — 1 in the Taylor expansion of log(l + x) about 
x^O. 

In passing, without proof, here is a useful test to check convergence of 
alternating series. 

Theorem 1. (Alternating series test) Consider the series 

oo 
j=0 

where aj > 0. // (aj) converges to zero (as a sequence), then the series is 
convergent. 

The main problem with conditionally convergent series is that if the terms 
are rearranged, then the series may converge to a different limit. The "safe 
zone" for handling infinite sums as if they were finite is when convergence is 
absolute. 

Theorem 2. Let f : Z"*" i— > Z"*" be a bijection, i.e., f is a rearrangement of the 
nonnegative integers. Consider a series X^°lo%- If this series is absolutely 
convergent, then 

oo oo 
3=0 j=0 

Here is what usually happens when the assumption of absolute conver- 
gence is not satisfied. 
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CHAPTER 1. SERIES AND SEQUENCES 



Example 6. Consider again 

1111 

which as we have seen equals log(2) — (1 — 1/2) = log(2) — 1/2 = .193147. . .. 
We can rearrange the terms of the series by assigning two negative terms for 
each positive term: 

11111 1 1 
3~4~6^5~8~10^7^'" 

This series is also convergent, but happens to converge to (log(2) — l)/2 = 
-.153426. . .. 

Other operations that can be safely performed on absolutely convergent 
series are passing absolute values inside the sum, and exchanging sums. 
Again, complications arise if the series is only conditionally convergent. (See 
Tao, Analysis 1, for counter-examples.) 

Theorem 3. The following operations are legitimate for absolutely conver- 
gent series. 

• Passing absolute values inside sums: 



j=0 j=Q 



'J\- 



Swapping sums: 

OO OO OO CX) 

^^'^ = 

j=0 fe=0 fe=0 3=0 



Note in passing that the same is true for integrals of unbounded integrands 
or integrals over unbounded domains: they need to be absolutely convergent 
(integrability of the absolute value of the function) for the integral swap to be 
legitimate. This is the content of Fubini's theorem. Again, there are striking 
counter-examples when the integrals are not absolutely convergent and the 
swap doesn't work (See Tao, Analysis I). 
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1.2 The big-O notation 

Here are a few useful pieces of notation for comparing growth or decay of 
sequences, used extensively by numerical analysts. They are called the big- 
O, little-o, and big-Theta notations. Big-0 is used much more often than 
the other two. They occur when comparing decay rates of truncation errors, 
and runtimes of algorithms. 

Definition 4. Consider two nonzero sequences fn and (?„ for n = 0, 1, 2, . . .. 
We write fn = 0{gn) when there exists C > such that \ fn\ < C'lgi^j. 
We write fn = o{gn) when fn/dn —^0 as n — > oo. 
We write fn = Q{gn) when fn = 0{gn) and = 0{fn)- 

Examples: 

• fn = i^"^ and gn = n^: we have = 0{n^) and = o{n^) but 

^ 0(n3). 

• /" = ^ ^"^^ 9n = we have /„ = 0{gn) and /„ = Q{gn), but 
fn + o{gn). 

• Exponentials always dominate polynomials: n" = o{e'^'^) whatever a > 
and 6 > 0. 

• Conversely, e^'^ — o(n~"). 

Out loud, we can read the expression = 0{gn) as "/„ in on the order 

of gn" ■ 

The same notations can be used to compare sequences indexed by a pa- 
rameter that goes to zero, such as (typically) the grid spacing h. The defini- 
tion above is simply adapted by letting /i — > rather than n — > oo. 

Examples: 

• f{h) — and g{h) — h^: this time we have g{h) — 0{f{h)) and 
g{h) — o{f{h)) when the limit of interest is /i — > 0. 

• Powers of h don't converge to zero nearly as fast as this exponential: 
^a/h ^ whatever a > and 6 > 0. 

Similarly, we may wish to compare functions / and of a continuous 
variable x as either x — > oo or x ^ 0; the definition is again modified in the 
obvious way. Whenever a O(-) or o(-) is written, some underlying limit is 
understood. 
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